To deal with the problems in line segment matching due to occlusion, fragmentation and inaccurate extraction of line segment endpoints, especially when the criteria of "most similar matching" was taken in "multiple-to-multiple" matching which may lead to fateful true correspondences lost, a line segment matching algorithm based on geometric constraints and 0-1 programming was proposed. Firstly, a candidate matching set was initially computed based on the spatial adjacency after correcting the vedio frames. Secondly, the multiple geometric constraints, such as epipolar constraint, homography matrix and point-to-line adjacency, were employed to remove the false positive correspondences. Then, the matching problem was modeled into a large scale 0-1 programming. Finally, a two-stage method based on grouping strategy was designed to solve this problem, so as to realize the "one to one" exact matching of line segment feature. The experimental results show that, compared with LS (Line Sigature) and LJL (Line-Junction-Line) methods, the propsed method has a similar performance in correct matching ratio but a larger matching amount over 60% and 11%, respectively. The proposed method can fulfill the line segment matching across vedio frames, which lays the foundation for line-based visual SLAM (Simultaneously Localization and Mapping).